PSS : Add `fs` and `inmem` state storage implementations to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

SarahFrench · 2025-10-20T19:10:04Z

This PR enables E2E testing with pluggable state storage:

Implements PSS support in the grpcwrap package, which is needed to allow provider-like code in Core to behave like it's a real separate provider plugin.
Implements an inmem and fs state store in the v6 version of terraform-provider-simple
- Note: the filesystem PSS implementation doesn't include locking. In discussions we agreed that there was nothing to gain by implementing locking, as our E2E tests don't include more than one client.
- If locking-specific testing is added in future then locking can be implemented then.
Adds E2E tests using the two PSS implementations, showing an init followed by an apply command.

Target Release

N/A

Rollback Plan

If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

This change is user-facing and I added a changelog entry.
This change is not user-facing.

internal/builtin/providers/terraform/provider.go

internal/provider-simple-v6/state_store_inmem.go

.github/workflows/checks.yml

…tations, pass through requests to matching store based on type name.

… implementation; add embeddable unimplementedProviderInterface struct

…orm provider. Stop people using it unless TF_ACC is set.

…pace state already.

…ore.

…n terraform provider

…they can be reused.

…ther work to allow PSS to be used with other commands.

…state storage.

This provider is not accessible to users so it's unnecessary.

Seeing as I was in this part of the code during this branch's work

SarahFrench · 2025-11-06T10:52:48Z

internal/backend/pluggable/pluggable.go

+		Capabilities: providers.StateStoreClientCapabilities{
+			ChunkSize: DefaultStateStoreChunkSize,
+		},


I realised that we never sent the default chunk size data from Core to the provider at the start of chunk size negotiation 😬

…hod.

radeksimko

Looks pretty good, I left mostly minor comments and questions in-line but stand ready to approve once answered/addressed.

internal/command/e2etest/primary_test.go

internal/grpcwrap/provider6.go

internal/provider-simple-v6/state_store_fs.go

internal/provider-simple-v6/state_store_fs_test.go

internal/provider-simple-v6/state_store_inmem.go

…nts enabled, avoiding use of global variables. Avoiding global variables also means we don't need to set TF_TEST_EXPERIMENTS in the GitHub Actions workflow.

#37790 (comment)

…hod is implemented

Co-authored-by: Radek Simko <[email protected]>

… "state ids"

Related to 864dd77

radeksimko

LGTM, looking forward to using this to build out more E2E tests! 👍🏻

SarahFrench added the no-changelog-needed Add this to your PR if the change does not require a changelog entry label Oct 20, 2025

SarahFrench commented Oct 20, 2025

View reviewed changes

internal/builtin/providers/terraform/provider.go Outdated Show resolved Hide resolved

internal/provider-simple-v6/state_store_inmem.go Show resolved Hide resolved

SarahFrench changed the title ~~PSS : Add inmem state storage implementation to the builtin terraform provider~~ PSS : Add inmem state storage implementation to the builtin terraform provider, update grpcwrap package, use PSS implementation in E2E test Oct 24, 2025

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch from a4887d9 to ef6dc39 Compare October 24, 2025 11:04

SarahFrench commented Oct 24, 2025

View reviewed changes

.github/workflows/checks.yml Outdated Show resolved Hide resolved

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch 2 times, most recently from 0f01ff6 to 2fadb1e Compare October 24, 2025 11:38

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch from 727ae19 to 8f77b19 Compare November 3, 2025 11:05

SarahFrench added 21 commits November 6, 2025 09:59

Allow the builtin terraform provider to contain multiple PSS implemen…

60e8997

…tations, pass through requests to matching store based on type name.

Make it easier to use provider.Interface as interface for state store…

306d896

… implementation; add embeddable unimplementedProviderInterface struct

Add initial implementation of inmem state store to the builtin terraf…

9f104fa

…orm provider. Stop people using it unless TF_ACC is set.

Add method for getting an inmem store instance that has default works…

b79de8d

…pace state already.

Copy some tests from the inmem backend and use for the inmem state st…

f54f95b

…ore.

Simplify code to avoid map of state stores, for now

a3ffebd

WIP simple

c4e0274

Implement inmem state store in provider-simple-v6, remove from builti…

759df39

…n terraform provider

Move PSS chunking-related constants into the pluggable package, so …

c6cea63

…they can be reused.

Implement PSS-related methods in grpcwrap package

320fa96

Update e2e test - works as expected but is blocked at apply step by o…

992eb53

…ther work to allow PSS to be used with other commands.

Fix issues in test fixture, rename it

eeaaac8

Update test - remove steps that are impossible to perform with inmem …

099b206

…state storage.

Stop gating the inMem state store behind TF_ACC in the simple6 provider

40b9930

This provider is not accessible to users so it's unnecessary.

Skip E2E test for PSS unless experiments enabled

4705e52

Enable experiments in E2E tests in automation

bd0fa59

Add missing space to error message text

5f0581f

Fix TF_TEST_EXPERIMENT => TF_TEST_EXPERIMENTS

b986709

Ensure state stores are configured with a suggested chunk size from Core

2146307

Small changes to inmem E2E test

0cc10c5

Add filesystem state store (no locking) and E2E test

9979fe1

Replace ioutil.ReadDir with os.ReadDir

316d5b5

Seeing as I was in this part of the code during this branch's work

SarahFrench commented Nov 6, 2025

View reviewed changes

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch from 8f77b19 to 316d5b5 Compare November 6, 2025 12:40

SarahFrench marked this pull request as ready for review November 6, 2025 15:11

SarahFrench requested a review from a team as a code owner November 6, 2025 15:11

fix: Pass through id of state to delete in grpcwrap's DeleteState met…

9088fe5

…hod.

SarahFrench mentioned this pull request Nov 7, 2025

PSS: Update the workspace new subcommand to work with PSS, add E2E and integration tests for using workspace commands with PSS. #37855

Merged

3 tasks

radeksimko reviewed Nov 10, 2025

View reviewed changes

SarahFrench and others added 7 commits November 11, 2025 12:28

refactor: make tests build their own Terraform binaries with experime…

9bfecfb

…nts enabled, avoiding use of global variables. Avoiding global variables also means we don't need to set TF_TEST_EXPERIMENTS in the GitHub Actions workflow.

fix: don't check if file exists before trying to read from it

864dd77

#37790 (comment)

refactor: change how the inmem store's ValidateStateStoreConfig met…

33d6670

…hod is implemented

chore: update code comment

c038710

Co-authored-by: Radek Simko <[email protected]>

fix: remove "PSS" from grpcwrap error

094607f

Co-authored-by: Radek Simko <[email protected]>

refactor: Reduce use of the word "workspace" in favour of "states" or…

ba3b416

… "state ids"

fix: Protect against empty file

247bb96

Related to 864dd77

SarahFrench requested a review from radeksimko November 11, 2025 14:43

radeksimko approved these changes Nov 11, 2025

View reviewed changes

SarahFrench merged commit f2818db into main Nov 11, 2025
9 checks passed

SarahFrench deleted the pss/add-builtin-inmem-state-storage-implementation branch November 11, 2025 15:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PSS : Add `fs` and `inmem` state storage implementations to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

PSS : Add `fs` and `inmem` state storage implementations to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

Uh oh!

SarahFrench commented Oct 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SarahFrench Nov 6, 2025

Uh oh!

radeksimko left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

radeksimko left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PSS : Add fs and inmem state storage implementations to the builtin simplev6 provider, update grpcwrap package, use PSS implementation in E2E test #37790

PSS : Add fs and inmem state storage implementations to the builtin simplev6 provider, update grpcwrap package, use PSS implementation in E2E test #37790

Uh oh!

Conversation

SarahFrench commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Target Release

Rollback Plan

Changes to Security Controls

CHANGELOG entry

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SarahFrench Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

radeksimko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

radeksimko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PSS : Add `fs` and `inmem` state storage implementations to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

PSS : Add `fs` and `inmem` state storage implementations to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

SarahFrench commented Oct 20, 2025 •

edited

Loading